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•° Testing 



Goal 

• Does program P obey 
specification S? 

what is P? 
what is S? 


Slides originally by Ken Wong 

Images reproduced in these slides have been included under 
section 29 of the Copyright Act, as fair dealing for research, 
private study, criticism, or review. Further distribution or 
uses may infringe copyright. 


Approaches 

• Reasoning about the state 
model for P: 

typically a huge number of states 
every practical technique must be 
inaccurate 

could abstract states 
could sample states 
or both 


Approaches 

• Abstraction: 

° often used in static software 
analysis techniques 
e.g., model checking P for some 
specific S 

techniques often pessimistically 
inaccurate 

may report P is faulty when P is 
correct 


Approaches 

• Sampling: 

often used in dynamic analysis techniques 
|]e.g., testing, profiling 

techniques often optimistically inaccurate 
may report P is correct when P is faulty 

testing drives P through a sampling of states, 
but the samples may not generalize to actual 
situations 
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State-Based Testing 

• Steps: 

set up software into a known state 

□ e.g., initialize variables 

trigger transitions to cause state changes 
e.g., call methods to change variables 

verify the actual arrived state is expected 
e.g., set if actual values in variables meet 
expectations 


Software Defects 

• Some terms: 

° human errors can lead to faults in work 
products, which may cause failures 
when running the software 

can try to find faults through testing, 
reviews, proof, model checking, code 
analysis, etc. 

°some avoid the term bug, since it 
implies something wandered into the 
code 
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Failure 

• AT&T failure (1990): 

114 switching nodes of their long 
distance system crashed 
the outage lasted for 9 h, 

70 million calls went uncompleted 

• Reason: 

if a node crashes, it tells neighboring 
nodes to reroute traffic around it 
a bug in handling this message caused 
the receiving node to also crash, etc. 



Fault in Code 


• Root cause: 


do { 


switch (...) { 
case 

if (...) { 


break ; 
} else { 


} 

} while (...) ; 


after expensive testing phase, 
a small change was made 
without again retesting 


Examples of Defects 

• Actual behavior differing from expected: 
algorithmic 

Qcode logic does not produce the proper output 
overload 

data structure unexpectedly filled to capacity 
performance 

violates service level agreement 
accuracy 

calculated result not to the desired level of accuracy 
timing 

race condition in coordinating concurrent processes 
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Why Test? 

• Goals: 

verification 

□ check that requirements are satisfied 
not only to confirm normal behavior 
find problems to refute that the program is 
correct 

establish due diligence 
evidence in case of product liability litigation 
avoid regression 

prevent previous problems from reoccurring 
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Regression Testing 

• Goal: 

to avoid breaking things that should 
work 

collect, reuse, and re-run automated test 
cases 

do regression test after a change or fix 
Q re-run tests to check whether previously 
passing tests of the system now fail 
D e.g., old defect somehow became unfixed 


Limits of Testing 

• Issues: 

a program cannot be tested completely 
[jtoo many inputs and path combinations to cover 
testing cannot find all defects 
cannot show their absence, just their presence 

challenging 

testing may be expensive and frustrating 
test code itself could add its own defects 
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Black Box Testing 

• Example test cases: 

be systematic about what to 
test, 

not knowing the internal code 



Addends 

Sum 

Description (also check commutative) 

2 

3 

5 

something simple 

99 

99 

198 

large positive pair 

99 

-14 

85 

large positive plus negative 

99 

16 

115 

large positive plus positive 

-99 

-99 

-198 

large negative pair 

-99 

-14 

-113 

large negative plus negative 

-99 

16 

-83 

large negative plus positive 

-99 

99 

0 

large positive plus large negative 

9 

9 

18 

largest single digit positive pair 


Black Box Testing 

•Tips: 

avoid redundant tests 
Gtoo easy to keep adding meaningless 
extra tests 

°determine equivalence classes of 
tests 


partitioning of test cases 


Black Box Testing 

•Equivalence classes: 

each test inside an equivalence 
class checks the "same thing" 
if a test inside the class will catch a 
defect, the other tests probably 
also will 

if a test inside the class will not 
catch a defect, the other tests 
probably also will not 
keep only a few tests in each class, 
as representatives 



Black Box Testing 

• Example test cases: 

° guessing at internal algorithm or 


0 

representation 

Addends Sum Description (also check commutative) 

0 0 all zero special case 

4/3 

$2 

Addends 

2 

$2 

0 

23 

23 

zero plus positive 

+ 5 

3 

-78 

0 

-78 

negative plus zero 

(9) 

9 

127 

127 

254 

max signed bytes 
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-128 

127 

-1 

min and max signed bytes 

1 

1 

-128 

-128 

-256 

min signed bytes 

0 

0 

2147483647 

2147483647 


max signed integers 

<tab> 

<tab> 

-2147483648 

2147483647 

-1 

min and max signed integers 

1.2 
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Black Box Testing 

• Example test cases: 

data input from fields in user 
interface 


min signed integers 


expression 
currency symbols 
plus sign 

parentheses around negatives 
lower case letter I 
upper case letter O 
no input 
decimal 

invalid characters 


-2147483648 


-2147483648 
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Black Box Testing 

• Example test cases: 

and even more user interface 
explorations 

editing with delete, backspace, 

cursor keys, etc. 

using FI, escape, and control 

characters 

vary timing of data entry 


Defect Tracking 

• Typically, for each reported defect: 
identification 
□ ID 

program and version 
° classification 

kind of defect (e.g., code or documentation) 
severity (e.g., minor, major, critical) 
description 
issue 

how to reproduce 
suggested fix (optional) 
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Defect Tracking 

• For each reported defect: 

c progress 

□ status (open or closed) 
resolution (e.g., pending, fixed, 
irreproducible, deferred, as designed, 
unfixable) 

° involved person 
reported by and when 
assigned to and when 
resolved by and when 
verified by and when 


Testing Strategies 

• Big-bang strategy: 

test thoroughly only after the 
whole system is put together 

° pro? 

D "project almost finished, only 
testing left" 

cons 

D hard to pinpoint the cause of a 
failure 
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Testing Strategies 

• Top-down incremental strategy: 

implement/test the highest-level modules first 
provide stubs for lower-level functionality not yet 
implemented 

higher-level modules are the test drivers 



Testing Techniques 

•Creating good tests: 
test every error message 
error-handling code tends to be 
weaker 


• Bottom-up incremental strategy: 
o implement/test the lowest-level modules first 
need to write test drivers 


test under other configurations 
D programmers are biased to their 
own setup 


* Design for Testing 
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Good Software Design 

• Want software to be flexible: 

easy to change to respond to new needs 
easy to understand 

easy to extend, without exploding complexity 


• Want software to be testable: 
easy to construct the units 
easy to set up units into desired state 
easy to drive code and witness effects 
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Example Bad Design 1 

• /*★ 

* Process photo album requests, 

* parse user preferences, 

* apply image transformations, 

* assemble images into albums, 

* deliver results to users 
*/ 

public class PhotoAlbumServer { 

... // lots of code 

} 


Example Bad Design 1 

• Poor flexibility: 

° difficult to extract and reuse parts 
° complex to add new features 
° instance variables are "global" 

• Poor testability: 

°only end-to-end testing possible 
c need golden results files for every 
combination of preference settings 
and image transformations 
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Improved Design 1 

• Use separation of concerns: 

° RequestHandler class 
° UserPreferencesReader class 
° UserPreferencesParser cass 
ImageEffect class 
° ImageTransformer class 


Improved Design 1 

• Better flexibility: 

uses object-oriented design 
c easier to understand smaller, 
separate units 

• Better testability: 

more focused tests of each unit 
test fixtures easier to provide for 
each unit 

easier to check results 
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Forming Dependencies 

• public class ExampleService { 

private DataSource theDataSource; 



"Dependency Injection" 

• public class ExampleService { 

private DataSource theDataSource; 


public ExampleService (...) { 

theDataSource = new DataSource ( ... ) ; 


public ExampleService( 

DataSource aDataSource ) { 


theDataSource = aDataSource; 


public void doService() { 


... = theDataSource . getlnfo () ; 


public void doService() { 


one approach is that the class 
makes what it depends on 


ExampleService 


DataSource 
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... = theDataSource . getlnfo () ; 


alternatively, 
construct what this 
class depends on 
outside the class 


System Assembly 




System Assembly without Dl 

• public class CellPhone { 

public CellPhone() { 

radio = new GSMRadio(); 
inputDevice = new KeyPad(); 


} 

} 

public class GSMRadio { 

public GSMRadio() { 

chipset = new QualcommGSMChipset(); 
antenna = new FractusQuadBandAntenna(); 

} 

} 


CellPhone phone = new CellPhone(); 
// fully assembled 
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System Assembly without Dl 

• Poor flexibility: 

difficult to change and plug in parts 

[]for different radio, different input device, etc. 


• Poor testability: 

can't supply test versions of parts 
stuck with given parts 
entire aggregate is constructed 
could be expensive 


System Assembly with Dl 

• public class CellPhone { 

public CellPhone( Radio radio, 

InputDevice inputDevice ) { 

this.radio = radio; 

this.inputDevice = inputDevice; 


} 

public class GSMRadio { 

public GSMRadio( Chipset chipset, 
Antenna antenna ) { 

this.chipset = chipset; 
this.antenna = antenna; 

} 

} 
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System Assembly with Dl 


• // in some high-level class 

CellPhone phone = new CellPhone( 
new GSMRadio( 

new QualcommGSMChipset(), 
new FractusQuadBandAntenna() 

), 

new Keypad () separates out 

); "dependency resolution 

from the constituent 
classes 



e.g., QualcommGSMChipset e.g., FractusQuadBandAntenna 

extends Chipset extends Antenna 


could have other subclasses beyond these examples 
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System Assembly with Dl 



the bottom-up assembly process instantiates 
the children and inserts them into the parents 



Example Bad Design 2 

• public class User { 

private Preferences prefs; 

public User( File prefFile ) { 

prefs = parseFile( prefFile ); 


} 

public void doSomething() { 

... // use prefs 

} 

private Preferences parseFile( File prefFile ) { 

aPrefs = new Preferences ( ... ) ; 

... // setup prefs 
return aPrefs; 

} 

} 
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Example Bad Design 2 

• Poor flexibility: 

changing preferences requires 
changing User 
file format changes 
difficult to reuse User 
embedded preference file reading and 
parsing 


• Poor testability: 

tests that deal with files are slow 
need test file for each preference 
combination 


Improved Design 2 

• class User { 

private Preferences prefs; 

public User( Preferences prefs ) { dependency 
this, prefs = prefs; injection 


) 

public void doSomething() { 
... // use prefs 

} 


} 
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Improved Design 2 

• Better flexibility: 

c no change to User if file format 
changes 

0 preferences not limited to be 
made from files 

• Better testability: 

°can run fast 

Dpass in mock or fake Preferences 
object 



"Mock Object" 

• public class UserTest { 

public void testdoSomething() { 

// MockPreferences extends Preferences, 

// but is overridden with canned settings 
// (no test preference file needed) 

MockPreferences mockPrefs = 
new MockPreferences (); 


User aUser = new User( mockPrefs ); 


aUser.doSomething(); 


} 


mockPrefs.AssertNoChange(); 
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Example Bad Design 3 

• Situation: 

c many pieces of information are 
needed by classes throughout 
the system 



Example Bad Design 3 

•Typical approaches: 

consumers get the data they 
need ... 


° but each class needs just one or 
a few items 

°how to provide this information 
to the consumers? 


°make the data global, 

°pass around a context object, or 
° put the data in widely known and 
used classes 
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Example Bad Design 3 

• public class Account { 

public Account( User user ) { 

this.country = 

user. getPreferences() .getLocation() .getCountry(); 

} 

} 


Example Bad Design 3 

• Poor flexibility: 

° method parameters do not show 
what the method really needs 
code "locks in" the structure it 
walks 

• Poor testability: 

test needs to recreate this 
structure ... 
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Example Bad Design 3 

• public void testSomethingForAccount() { 

// set up for test 

Country country = new Country( "Canada" ); 

Location location = new Location(); 
location.setCountry( country ); 

Preferences prefs = new Preferences(); 
prefs.setLocation( location ); 

User user = new User( prefs ); 

Account account = new Account( user ); 

... // test Canadian account 

} 

test code should be simple (less likely to have defects) 


Improved Design 3 


public void testSomethingForAccount() { 

Country country = new Country( "Canada" ) ; 

// redesigned constructor 
// (requires only what is needed) 

Account account = new Account( country ); 

... // test Canadian account 

} 
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•° Test-Driven 
Development 


Automated Testing 

• Purpose: 

< write software to help test software 

automation essential to test-driven 
development and refactoring 

• Limitations: 

< manual testing still need to observe 
certain problems 

e.g., strange noises from the speaker, 
flickering graphics 
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Automated Testing 

• A good automated unit test: 
is simple to write and understand 
0 reduces the chance of defects in the test code 
runs quickly 

so it can be re-run frequently while developing 
is isolated 

could run multiple unit tests in parallel 
shows exactly what went wrong if it fails 
reduce time spent in a debugger 


Automated Testing 

• Quote: 

““Whenever you are tempted to 
type something into a print 
statement or a debugger 
expression, write it as a test 
instead." 


— Martin Fowler 


“Way of Testivus" 
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“Way of Testivus" 


• “Think of code and tests as one 

When writing the code, think of the tests. 
When writing the tests, think of the code. 

When you think of code and tests as one, 
testing is easy and the code is beautiful." 

— Alberto Savoia 



• "Best time to test is when the code is fresh 

Your code is like clay. 

When it's fresh, it’s soft and malleable. 

As it ages, it becomes hard and brittle. 


If you write tests when the code is fresh and easy 

to change, testing will be easy, 

and both the code and the tests will be strong." 


— Alberto Savoia 


Test-Driven Development 

• Idea: 

if testing is so useful, let's write the 
tests first 

° these automated tests capture code¬ 
level requirements to be satisfied 

°once code is written so that these tests 
pass, then these requirements are 
considered to be met 


traditional 

development 
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test-first or 
test-driven 
development 



JUnit Framework 

• Usage: 

for each class foo 
to be tested, 
implement a 
subclass named 
FooTest Of TestCase 
in the same 


Test 

nin(TestResuft) ^ 


E923E^ 


TestCase 


run(TestResult) 


TestSuite 


run(TestResult) 
aJjEWC est) 
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JUnit Framework 


• FooTest class has: 

test objects that may bis named test Something 

used in the test 

methods 


may initialize more specific test 
setup () method to objects 
initialize the test 
objects (or fixture) 

before each test for the test objects, calls the 

method is run method in foo to be tested 


tearDown () method to checks the results against what is 
clean up the fixture expected using assertion 

afterwards statements 


JUnit Framework 

• Example test code: 


public class NumberTest extends TestCase { 
private Number aNumber; 
private Number anotherNumber; 

protected void setup() { 

aNumber = new Number(2); 
anotherNumber = new Number(3); 

} 

// check that value-based equality works 
public void testEquals0 { 

Assert.assertTrue(!aNumber.equals(null)); 

Assert.assertEquals(aNumber, aNumber); 

Assert.assertEquals(aNumber, new Number(2)); 
Assert.assertTrue(!aNumber.equals(anotherNumber)); 

} 
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JUnit Framework 


In the Application 


• Example test code: 


• Example functional code: 


public void testAdd() { 

// more test data 

Number expected = new Number(5); 

// test Number.add method 

Number result = aNumber.add(anotherNumber); 
// check the result 

Assert.assertTrue(expected.equals(result)) ; 


• Assert static methods: 

http://junit.sourceforge.net/ 
j avadoc/junit/framework/As sert.html 


public class Number { 
private int value; 

public boolean equals(Object anObject) { 
if (anObject instanceof Number) { 

Number aNumber = (Number)anObject; 
return aNumber.value == this.value; 

} 

return false; 

} 
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In the Application 


• Issue: 

What methods 
should be tested 
with JUnit? 


Approach: 

write JUnit tests for 
methods of the 
application model that 
have side effects (i.e., 
not getter methods) 


use assertions on the 
output of getter 
methods to check that 
constructors and 
setter methods 
worked properly 



JUnit Framework 


Test 


run(TestResult) 




Collecting Parameter 


TestCase 


TestSuite 


run(TestResult) 
runTestQ 
setUp 0 
tearDownQ 


run(TestResult) 
aMfesKT est) 

o—- 



fName 







runTestQ 
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More Information 

• Books: 

° Test-Driven Development 

□ K. Beck 

Addison-Wesley, 2003 



More Information 

• Books: 

Testing Computer Software 
D C. Kaner, J. Falk, H. Q. Nguyen 
Wiley, 1999 


° Lessons Learned in Software 
Testing 

C. Kaner, J. Bach, B. Pettichord 
Wiley, 2002 


'2 


More Information 

• Links: 

Cause of AT&T Network Failure 
http://catless.ncl.ac.uk/Risks/9.62.ht 
ml#subj2 



More Information 

• Links: 

° Flexible Design? Testable Design? 
You Don't Have to Choose! 

R. Rufer and T. Bialik 


° History's Worst Software Bugs 
http ://www. wi red. com/softwa re/cool 
a pps/news/2005/11/69355 


The Way of Testivus 

D http://www.agitar.com/downloads/TheWayO 
fTestivus.pdf 

JUnit Resources for Test-Driven 
Development 
D http://www.junit.org/ 


More Information 


JUnit framework 

http://www.junit.org/ 


Test Infected 


http://junit.sourceforge.net/doc/testinfected/testing.htm 


